feat(#182): add CI path filter coverage gap check to review agent by fullsend-ai-coder[bot] · Pull Request #216 · guyoron1/fullsend

fullsend-ai-coder · 2026-07-08T12:15:30Z

Add a new review heuristic that cross-references PR-modified files against CI workflow path filters. When a modified file is exercised by a path-filtered CI workflow but not included in the workflow's paths trigger, the correctness sub-agent now flags it as a medium-severity ci-coverage-gap finding.

Changes:

correctness sub-agent: add "CI path filter coverage gap" section
with detailed procedure, severity, description format, remediation
guidance, and edge case handling
code-review skill: add ci-coverage-gap bullet under the Correctness
dimension for standalone (non-orchestrator) review mode
pr-review orchestrator: add ci-coverage-gap to the correctness
dimension category mapping table for prior-finding routing

The issue also recommended adding internal/harness/** to the functional-tests.yml path filter. This repo does not have a functional-tests.yml workflow; the equivalent e2e.yml already uses
**/*.go which covers all Go files including internal/harness/.
No path filter change is needed here.

Note: pre-commit could not run due to sandbox infrastructure error (shellcheck download blocked by network policy). The post-script runs pre-commit authoritatively on the runner.

Closes #182

Post-script verification

Branch is not main/master (agent/182-ci-path-filter-check)
Secret scan passed (gitleaks — f96750babbed5ada406a9ae04e8068449701d9c7..HEAD)
Pre-commit hooks passed (authoritative run on runner)
Tests ran inside sandbox

Add a new review heuristic that cross-references PR-modified files against CI workflow path filters. When a modified file is exercised by a path-filtered CI workflow but not included in the workflow's paths trigger, the correctness sub-agent now flags it as a medium-severity ci-coverage-gap finding. Changes: - correctness sub-agent: add "CI path filter coverage gap" section with detailed procedure, severity, description format, remediation guidance, and edge case handling - code-review skill: add ci-coverage-gap bullet under the Correctness dimension for standalone (non-orchestrator) review mode - pr-review orchestrator: add ci-coverage-gap to the correctness dimension category mapping table for prior-finding routing The issue also recommended adding internal/harness/** to the functional-tests.yml path filter. This repo does not have a functional-tests.yml workflow; the equivalent e2e.yml already uses **/*.go which covers all Go files including internal/harness/. No path filter change is needed here. Note: pre-commit could not run due to sandbox infrastructure error (shellcheck download blocked by network policy). The post-script runs pre-commit authoritatively on the runner. Closes #182

github-actions · 2026-07-08T12:20:30Z

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

fullsend-ai-review · 2026-07-08T12:37:04Z

Review

Findings

Low

[design-direction] internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md:50 — The new CI path filter coverage gap check adds a 46-line procedural algorithm to the correctness sub-agent. While the procedure is more structured than most sub-agent heuristics (numbered steps, explicit edge cases), it is comparable in procedural detail to existing patterns in other sub-agents — the security sub-agent includes a similarly detailed 4-step verification methodology, and the intent-coherence sub-agent includes multi-step revert verification logic. If the check produces inconsistent results in practice, consider extracting it to a pre-review script.
[design-smell] internal/scaffold/fullsend-repo/skills/pr-review/SKILL.md:13 — The pr-review skill explicitly acknowledges its departure from ADR-0018 and documents mitigations (fixed sub-agent roster, structured context packages, deterministic post-processing). The absence of a formal superseding ADR creates architectural ambiguity about whether LLM-based orchestration is authorized for review workflows. This is a pre-existing condition not introduced by this PR. Consider writing a superseding ADR to formalize the current state.

Previous run

Review

Findings

Low

[design-direction] internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md:47 — The CI path filter coverage gap check is categorized under 'correctness' and placed in the correctness sub-agent. While the check is meta-level infrastructure validation rather than code correctness, it is defensible: the heuristic identifies situations where tests will silently not run, which is a test-adequacy concern that directly maps to the correctness dimension's responsibility for test integrity. Consider documenting the rationale for placing CI infrastructure checks under correctness.
[doc-style] internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md — The new CI path filter coverage gap section uses H3 subsection headers (Procedure, Severity, etc.) while some existing sections in correctness.md use bold-prefixed inline labels (e.g., 'Consumer completeness:'). However, the existing file already uses both patterns: shorter heuristics use bold labels, while the longer 'Technical documentation with correctness surface area' section uses an H3 header. The new section is similarly long and structured, so the H3 pattern is precedented within this file.

Previous run (2)

Review

Verdict: approve

Clean, well-scoped addition of a CI path filter coverage gap heuristic to the review agent's correctness dimension. All three modified files are internally consistent: the category name (ci-coverage-gap), severity (medium), and edge-case semantics match across the correctness sub-agent definition, the pr-review orchestrator's category mapping table, and the code-review skill summary.

Verification summary

Consumer completeness: The ci-coverage-gap category is added to the pr-review SKILL.md category-to-dimension mapping table. No other consumers require updating — ReviewFinding.Category is a free-form string in the Go code and JSON schema (no enum validation).
Internal consistency: Category name, severity, and edge cases are consistent across all three files. The code-review SKILL.md summary accurately condenses the full procedure from correctness.md.
PR body claim verified: This repository has no functional-tests.yml workflow. The e2e.yml workflow uses **/*.go in its paths filter, which covers all Go files including internal/harness/. No path filter change is needed in this repo.
Issue [ARCHIVED] Original issue 182 #182 scope match: The PR delivers exactly what the issue requests — a review heuristic that cross-references modified files against path-filtered CI workflow triggers. The three files changed (correctness sub-agent, pr-review orchestrator category mapping, code-review standalone skill) are the minimum necessary set.
Security: No injection patterns, secrets, Unicode steganography, or security-weakening guidance in the added text. The new instructions are legitimate correctness heuristic guidance.
paths-ignore semantics: The edge case stating "A file not in paths-ignore WILL trigger the workflow, so there is no gap" correctly reflects GitHub Actions behavior.

Findings

#	Severity	Category	File	Description
1	low	edge-case	`internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md`	Procedure step 2 says "check whether it matches any path-filtered workflow's `paths` list," but step 1 instructs the agent to also read workflows with `paths-ignore` filters. For `paths-ignore` workflows there is no `paths` list — only an ignore list. The edge-case section partially addresses this by explaining the non-gap case (file NOT in `paths-ignore` → workflow triggers → no gap), but is silent on the scenario where a file IS in `paths-ignore` (intentionally excluded) yet the workflow's test suite exercises that file's code. This is an unlikely scenario in practice (`paths-ignore` typically excludes docs, not source code), so the practical impact is minimal. A small wording clarification in step 2 — e.g., "check whether it matches the workflow's `paths` list (or is not excluded by `paths-ignore`)" — would close the gap.

Previous run (3)

Looks good to me

Previous run (4)

Review

Findings

Low

[design-direction] internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md:50 — The CI path filter coverage gap check extends the correctness dimension into CI infrastructure analysis (reading workflow YAML, evaluating glob patterns, inferring test coverage). While this falls within the existing "test adequacy" scope — detecting when tests are silently skipped is a correctness concern — the procedure is qualitatively different from the dimension's traditional focus on code logic and test assertions. Consider a brief scope note in the correctness sub-agent's Own field clarifying that test adequacy includes CI configuration that determines whether tests run.
[doc-style] internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md:50 — The new section uses ### heading level, which is correct — it is a peer to the existing ### Technical documentation with correctness surface area section. No change needed; noted for completeness.

Labels: PR adds a new review heuristic (CI path filter coverage gap check) to the review agent skills.

guyoron1 · 2026-07-08T14:03:47Z

/fs-review

guyoron1 · 2026-07-11T10:18:38Z

/fs-review

fullsend-ai-review · 2026-07-11T10:19:25Z

🤖 Finished Review · ✅ Success · Started 10:19 AM UTC · Completed 10:29 AM UTC
Commit: a2afa8b · View workflow run →

guyoron1 · 2026-07-11T10:32:26Z

/fs-review

fullsend-ai-review · 2026-07-11T10:33:10Z

🤖 Finished Review · ✅ Success · Started 10:33 AM UTC · Completed 10:45 AM UTC
Commit: a2afa8b · View workflow run →

guyoron1 · 2026-07-11T11:11:45Z

/fs-fix

fullsend-ai-coder · 2026-07-11T11:12:40Z

🤖 Finished Fix · ✅ Success · Started 11:12 AM UTC · Completed 11:23 AM UTC
Commit: a2afa8b · View workflow run →

Expand the correctness sub-agent's Own field parenthetical from 'test adequacy (are the right behaviors tested?)' to 'test adequacy (are the right behaviors tested, and will CI actually run them?)' — documenting why CI path filter coverage gap checks belong under the correctness dimension. Addresses review feedback on #216

fullsend-ai-coder · 2026-07-11T11:23:14Z

🔧 Fix agent — iteration 1 (human-triggered)

Addressed 1 of 2 review findings. The design-direction finding was fixed by expanding the correctness sub-agent's Own field to document that test adequacy includes CI configuration. The doc-style finding was informational (confirming the H3 pattern is correct) and required no code change.

Fixed (1):

document rationale for placing CI infrastructure checks under correctness (internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md): Expanded the correctness sub-agent Own field parenthetical from 'test adequacy (are the right behaviors tested?)' to 'test adequacy (are the right behaviors tested, and will CI actually run them?)' — documenting that test adequacy includes CI configuration that determines whether tests run, which is the rationale for placing the CI path filter coverage gap check under the correctness dimension.

Disagreed (1):

H3 subsection headers vs bold-prefixed inline labels style inconsistency: The reviewer noted that the H3 pattern is already precedented within the file for longer, structured sections. No style inconsistency exists — this was an informational observation confirming the current style is correct.

Tests: passed

_{Updated by fullsend fix agent}

fullsend-ai-review · 2026-07-11T11:27:40Z

🤖 Finished Review · ❌ Failure · Started 11:27 AM UTC · Completed 11:40 AM UTC
Commit: a2afa8b · View workflow run →

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge enhancement New feature or request labels Jul 8, 2026

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jul 8, 2026

This was referenced Jul 9, 2026

[RTK+Ponytail] Review agent should flag when PRs change code exercised by path-filtered CI tests without updating the filter #250

Closed

[ARCHIVED] Original issue 182 #182

Closed

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jul 11, 2026

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jul 11, 2026

guyoron1 closed this Jul 12, 2026

guyoron1 deleted the agent/182-ci-path-filter-check branch July 12, 2026 03:10

Conversation

fullsend-ai-coder Bot commented Jul 8, 2026

Post-script verification

Uh oh!

github-actions Bot commented Jul 8, 2026

E2E tests did not run

Uh oh!

fullsend-ai-review Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review

Findings

Low

Review

Findings

Low

Review

Verification summary

Findings

Review

Findings

Low

Uh oh!

This comment was marked as outdated.

Uh oh!

guyoron1 commented Jul 8, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

guyoron1 commented Jul 11, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

guyoron1 commented Jul 11, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

guyoron1 commented Jul 11, 2026

Uh oh!

fullsend-ai-coder Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-coder Bot commented Jul 11, 2026

🔧 Fix agent — iteration 1 (human-triggered)

Uh oh!

fullsend-ai-review Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fullsend-ai-review Bot commented Jul 8, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 11, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 11, 2026 •

edited

Loading

fullsend-ai-coder Bot commented Jul 11, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 11, 2026 •

edited

Loading